Method and device for processing and for searching for an object by signals corresponding to images

ABSTRACT

A method of representing an object appearing in a still or video image, by processing signals corresponding to the image, comprises deriving the peak values in CSS space for the object outline and applying a non-linear transformation to said peak values to arrive at a representation of the outline.

This application is a Divisional of U.S. application Ser. No. 11/822,833filed on Jul. 10, 2007, now U.S. Pat. No. 7,430,338 which is aDivisional of Ser. No. 11/156,751, filed Jun. 21, 2005, now U.S. Pat.No. 7,257,277 issued on Aug. 14, 2007, which is a Divisional of Ser. No.09/786,161, filed Mar. 2, 2001, now U.S. Pat. No. 6,931,154, issued onAug. 16, 2005; which claims priority to Japan Application PCT/JP00/04401filed on Jul. 3, 2000 under 35 U.S.C. § 120; based on UK 9915699.4 filedJul. 5, 1999 under 35 U.S.C. § 119; the entire content of which arehereby incorporated by reference.

The present invention relates to the representation of an objectappearing in a still or video image, such as an image stored in amultimedia database, especially for searching purposes, and to a methodand apparatus for searching for an object using such a representation.

In applications such as image or video libraries, it is desirable tohave an efficient representation and storage of the outline or shape ofobjects or parts of objects appearing in still or video images. A knowntechnique for shape-based indexing and retrieval uses Curvature ScaleSpace (CSS) representation. Details of the CSS representation can befound in the papers “Robust and Efficient Shape Indexing throughCurvature Scale Space” Proc. British Machine Vision conference, pp53-62, Edinburgh, UK, 1996 and “Indexing an Image Database by ShapeContent using Curvature Scale Space” Proc. IEE Colloquium on IntelligentDatabases, London 1996, both by F. Mokhtarian, S. Abbasi and J. Kittler,the contents of which are incorporated herein by reference.

The CSS representation uses a curvature function for the outline of theobject, starting from an arbitrary point on the outline. The curvaturefunction is studied as the outline shape is evolved by a series ofdeformations which smooth the shape. More specifically, the zerocrossings of the derivative of the curvature function convolved with afamily of Gaussian filters are computed. The zero crossings are plottedon a graph, known as the Curvature Scale Space, where the x-axis is thenormalised arc-length of the curve and the y-axis is the evolutionparameter, or in other words the parameter of the filter applied. Theplots on the graph form loops characteristic of the outline. Each convexor concave part of the object outline corresponds to a loop in the CSSimage. The co-ordinates of the peaks of the most prominent loops in theCSS image are used as a representation of the outline.

To search for objects in images stored in a database matching the shapeof an input object, the CSS representation of an input shape iscalculated. The similarity between an input shape and stored shapes isdetermined by comparing the position and height of the peaks in therespective CSS images using a matching algorithm.

It is desirable to find a representation of an object outline that isinvariant under changes of imaging conditions, such as changes in scale,rotations, some deformations of the outline and projectivetransformation. It is also desirable to represent a shape in a mannerthat is insensitive to shape variability within a broader class. Forexample, a representation of the object ‘car’ should be invariant to themodel and make of the car so that all car-like shapes can be easilyretrieved using the extracted feature representation.

Accordingly the present invention provides a method of representing anobject appearing in a still or video image, by processing signalscorresponding to the image, the method comprising deriving a pluralityof numerical values representing features appearing on the outline of anobject and applying a scalar or non-linear transformation to said valuesto arrive at a representation of the outline. Preferably, thetransformation is a non-linear transformation. Preferably, a CSSrepresentation is used, and preferably the peak height valuescorresponding to the evolution parameter are transformed.

It has been found that by applying a transformation, especially to CSSvalues, as in the invention, object retrieval performance is improved.

Embodiments of the present invention will be described with reference tothe accompanying drawings of which:

FIG. 1 is a block diagram of a video database system;

FIG. 2 is a drawing of an outline of an object;

FIG. 3 is a CSS representation of the outline of FIG. 2;

FIG. 4 is a diagram illustrating the representation of a shape;

FIG. 5 is a drawing of the shape of an object;

FIG. 6 is a CSS representation of the shape of FIG. 5;

FIG. 7 is a transformed representation of the shape of FIG. 5; and

FIG. 8 is a block diagram illustrating a searching method.

FIG. 1 shows a computerised video database system according to anembodiment of the invention. The system includes a control unit 2 in theform of a computer, a display unit 4 in the form of a monitor, apointing device 6 in the form of a mouse, an image database 8 includingstored still and video images and a descriptor database 10 storingdescriptors of objects or parts of objects appearing in images stored inthe image database 8.

A descriptor for the shape of each object of interest appearing in animage in the image database is derived by the control unit 2 and storedin the descriptor database 10. The control unit 2 derives thedescriptors operating under the control of a suitable programimplementing a method as described below.

Firstly, for a given object outline, a CSS representation of the outlineis derived. This is done using the known method as described in one ofthe papers mentioned above.

More specifically, the outline is expressed by a representationΨ={(x(u), y(u), uε[0,1]} where u is a normalized arc length parameter.

The outline is smoothed by convolving Ψ with an ID Gaussian kernel g(u,σ), and the curvature zero crossings of the evolving curve are examinedat σ charges. The zero crossing are identified using the followingexpression for the curvature:

${k\left( {u,\sigma} \right)} = \frac{{{X_{u}\left( {u,\sigma} \right)}{Y_{uu}\left( {u,\sigma} \right)}} - {{X_{uu}\left( {u,\sigma} \right)}{Y_{u}\left( {u,\sigma} \right)}}}{\left( {{X_{u}\left( {u,\sigma} \right)}^{2} + {Y_{u}\left( {u,\sigma} \right)}^{2}} \right)^{3/2}}$where X(u, σ) = x(u) * g(u, σ)  Y(u, σ) = y(u) * g(u, σ) andX_(u)(u, σ) = x(u) * g_(u)(u, σ)   X_(uu)(u, σ) = x(u) * g_(uu)(u, σ) 

Here * represents a convolution and the subscripts representderivatives.

The number of curvature zero crossings changes as σ changes, and when σis sufficiently high Ψ is a convex curve with no zero crossings.

The zero crossing points are plotted on a graph, known as the CSS imagespace. This results in a plurality of characteristic curves. The peaksof the characteristic curves are identified and the correspondingco-ordinates are extracted and stored. In general terms, this gives aset of n co-ordinate pairs [(x1,y1), (x2,y2), . . . (xn,yn)], where n isthe number of peaks, and xi is the arc-length position of the ith peakand yi is the peak height.

In this embodiment, a binomial filter with coefficients (¼, ½, ¼) isused as an approximation of a Gaussian filter with some reduction ofcomputational complexity. The reduction in computational complexityresults from convenient filter coefficients which can be efficientlyimplemented on a DSP or a general-purpose processor.

The peak values, or in other words, the y-component values for thepeaks, are then processed further. More specifically, the y values aretransformed using the transformation:y′=apow(y,b)+c  (1)

where pow(y,b) denotes y to the power b.

This results in a new set of peak values [(x1, y′1), (x2, y′2) . . .(xn, y′n)], which values are stored in the descriptor database as adescriptor of the outline.

As a specific example, the outline shown in FIG. 2 results in a CSSimage as shown in FIG. 3. Details of the co-ordinates of the peaks ofthe curves in the CSS image are given in Table 1 below.

TABLE 1 Peak Index X Y 1 0.124 123 2 0.68 548 3 0.22 2120 4 0.773 1001 50.901 678

The transformation given above is then applied, with a=6, b=0.5 and c=0.In other words, the square root of the original y value is taken andmultiplied by a constant. This results in the following values:

TABLE 2 Peak Index X Y 1 0.124 67 2 0.68 140 3 0.22 276 4 0.773 190 50.901 156

Here, the values are rounded to the nearest integer, but other roundingscan be used.

Another example is shown in FIG. 4.

FIG. 5 shows another example of an object shape, in this case a turtle.FIG. 6 shows the CSS peaks for the shape of FIG. 5. FIG. 7 shows thetransformed peaks of FIG. 6, using the transformation given in equation(1) above, with a=6.5, b=0.5 and c=0.

The stored descriptors are used for searching purposes. The userinitiates a search by drawing an object outline on the display using thepointing device (step 510). The control unit 2 then derives a CSSrepresentation of the input outline (step 520) and then applies thetransformation to the y values as described above (step 530). Theresulting descriptor of the input outline is then compared with eachstored descriptor in the descriptor database, known in the following asthe model descriptors, using a known matching procedure (step 540).

The matching comparison is carried out using a suitable algorithmresulting in a similarity measure for each descriptor in the database. Aknown matching algorithm such as described in the above-mentioned paperscan be used. That matching procedure is briefly described below.

Given two closed contour shapes, the image curve Ψi and the model curveΨm and their respective sets of peaks {(xi1,yi1),(xi2,yi2), . . .,(xin,yin)} and {(xm1,ym1), (xm2,ym2), . . . , (xmn,ymn)} the similaritymeasure is calculated. The similarity measure is defined as a total costof matching of peaks in the model into peaks in the image. The matchingwhich minimises the total cost is determined using a dynamicprogramming. The algorithm recursively matches the peaks from the modelto the peaks from the image and calculates the cost of each such match.Each model peak can be matched with only one image peak and each imagepeak can be matched with only one model peak. Some of the model and orimage peak may remain unmatched, and there is an additional penalty costfor each unmatched peak. Two peaks can be matched if their horizontaldistance is less then 0.2. The cost of a match is the length of thestraight line between the two matched peaks. The cost of an unmatchedpeak is its height.

In more detail the algorithm works by creating and expanding a tree-likestructure, where nodes correspond to matched peaks:

1. Create starting node consisting of the largest maximum of the image(xik, yik) and the largest maximum of the model (xir,yir).

2. For each remaining model peak which is within 80 percent of thelargest maximum of the image peaks create an additional starting node.

3. Initialise the cost of each starting node created in 1 and 2 to theabsolute difference of the y-coordinate of the image and model peakslinked by this node.

4. For each starting node in 3, compute the CSS shift parameter alpha,defined as the difference in the x (horizontal) coordinates of the modeland image peaks matched in this starting node. The shift parameter willbe different for each node.

5. For each starting node, create a list of model peaks and a list ofimage peaks. The list hold information which peaks are yet to bematched. For each starting node mark peaks matched in this node as“matched”, and all other peaks as “unmatched”.

6. Recursively expand a lowest cost node (starting from each nodecreated in steps 1-6 and following with its children nodes) until thecondition in point 8 is fulfilled. To expand a node use the followingprocedure:

7. Expanding a node:

-   -   If there is at least one image and one model peak left        unmatched:    -   select the largest scale image curve CSS maximum which is not        matched (xip,yip). Apply the starting node shift parameter        (computed in step 4) to map the selected maximum to the model        CSS image—now the selected peak has coordinates (xip-alpha,        yip). Locate the nearest model curve peak which is unmatched        (xms,yms). If the horizontal distance between the two peaks is        less then 0.2 (i.e: |xip-alpha−xms|<0.2), match the two peaks        and define the cost of the match as the length of the straight        line between the two peaks. Add the cost of the match to the        total cost of that node. Remove the matched peaks from the        respective lists by marking them as “matched”. If the horizontal        distance between the two peaks is greater than 0.2, the image        peak (xip,yip) cannot be matched. In that case add its height        yip to the total cost and remove only the peak (xip,yip) from        the image peak list by marking it as “matched”.    -   Otherwise (There are only image peaks or there are only model        peaks left unmatched):    -   Define the cost of the match as the height of the highest        unmatched image or model peak and remove that peak from the        list.

8. If after expanding a node in 7 there are no unmatched peaks in boththe image and model lists, the matching procedure is terminated. Thecost of this node is the similarity measure between the image and modelcurve. Otherwise, go to point 7 and expand the lowest cost node.

The above procedure is repeated with the image curve peaks and the modelcurve peaks swapped. The final matching value is the lower of the two.

As another example, for each position in the ordering, the distancebetween the input x value and the corresponding model x value and thedistance between the input y value and the corresponding model y valueare calculated. The total distance over all the positions is calculatedand the smaller the total distance, the closer the match. If the numberof peaks for the input and the model are different, the peak height forthe leftovers is included in the total distance.

The above steps are repeated for each model in the database (step 480).

The similarity measures resulting from the matching comparisons areordered (step 490) and the objects corresponding to the descriptorshaving similarity measures indicating the closest match (i.e. here thelowest similarity measures) are then displayed on the display unit 4 forthe user (step 500). The number of objects to be displayed can bepre-set or selected by the user.

An alternative embodiment will now be described. This embodiment is thesame as the previous embodiment, except that a different transformationis used. More specifically, the y values are transformed using thetransformation:y′=a ₀ +a ₁ y.

In other words, a linear, scaling, transformation is applied.

Here, a₀=41, a₁=0.19.

In a variation, a₀=0 and a₁=0.27.

Different values of a₀ and a₁ can be used as appropriate.

The searching and matching procedure is essentially as described in theprevious embodiment.

It has been found that applying a transformation, especially a lineartransformation involving scaling or a non-linear transformation, asdescribed above results in a descriptor which is less sensitive, eg tochanges of shape outline within an object class, which consequentlyresults in improved retrieval of objects.

In the embodiments described above, the transformation is applied to theCSS values before storing in the descriptor database 10. Alternatively,the CSS values can be stored in the database 10, and the transformationcarried out as part of the searching process, before the matchingprocedure is performed.

In the described embodiments, the transformations are applied to they-co-ordinate values. However, they may be applied to the x-co-ordinatevalues.

A system according to the invention may, for example, be provided in animage library. Alternatively, the databases may be sited remote from thecontrol unit of the system, connected to the control unit by a temporarylink such as a telephone line or by a network such as the internet. Theimage and descriptor databases may be provided, for example, inpermanent storage or on portable data storage media such as CD-ROMs orDVDs.

Components of the system as described may be provided in software orhardware form. Although the invention has been described in the form ofa computer system, it could be implemented in other forms, for exampleusing a dedicated chip.

Specific examples have been given of methods of representing a 2D shapeof an object, here, using CSS representation, and of methods forcalculating values representing similarities between two shapes but anysuitable such methods can be used.

The invention can also be used, for example, for matching images ofobjects for verification purposes, or for filtering.

1. A method of representing an object appearing in an image or asequence of images, by using at least one processor to process signalscorresponding to the image, the method comprising: receiving an outlineof the object; deriving, using said at least one processor, a pluralityof numerical values representing features appearing on the outline ofthe object using a binomial filter; deriving a descriptor representativeof the object based on the plurality of numerical values; and storingthe descriptor.
 2. A method as claimed in claim 1 wherein said binomialfilter has coefficients (¼, ½, ¼).
 3. A method as claimed in claim 1further comprising applying a scaling or non-linear transformation tosaid numerical values to arrive at a representation of the outline.
 4. Amethod of searching for an object in an image or sequence of imagesusing at least one processor to process signals corresponding to images,the method comprising inputting a query in the form of a two-dimensionaloutline, deriving, using said at least one processor, a descriptor ofsaid outline using a method as claimed in claim 1, obtaining adescriptor of objects in stored images derived using a method as claimedin claim 1 and comparing said query descriptor with each descriptor forstored object, and selecting and displaying at least one resultcorresponding to an image containing an object for which the comparisonindicates a degree of similarity between the query and said object. 5.An apparatus comprising: an image database storing an image or asequence of images; and a control unit configured to perform the methodof claim 1 or claim
 4. 6. A computer system comprising: an imagedatabase storing an image or sequence of images; and a control unitconfigured to perform the method of claims 1 or claim
 4. 7. Acomputer-readable storage medium storing computer-executable processsteps for implementing a method as claimed in claim 1 or claim 4.